Goodness of Fit: What Do We Really Want to Know?

The goodness-of-fit problem has recently attracted attention from the particle physics community. In modern particle experiments, one often performs an unbinned likelihood fit to data. The experimenter then needs to estimate how accurately the fit function approximates the observed distribution. A number of methods have been used to solve this problem in the past [1], and a number of methods have been recently proposed [2, 3] in the physics literature. For binned data, one typically applies a χ statistic to estimate the fit quality. Without discussing advantages and flaws of this approach, I would like to stress that the application of the χ statistic is limited. The χ test is neither capable nor expected to detect fit inefficiencies for all possible problems. This is a powerful and versatile tool but it should not be considered as the ultimate solution to every goodness-of-fit problem. There is no such popular method, an equivalent of the χ test, for unbinned data. The maximum likelihood value (MLV) test has been frequently used in practice but it often fails to provide a reasonable answer to the question at hand: how well are the data modelled by a certain density [4]? It is only natural that goodness-of-fit tests for small data samples are harder to design and less versatile than those for large samples. For small data samples, asymptotic approximations do not hold and the performance of every goodness-of-fit test needs to be studied carefully on realistic examples. Thus, the hope for a versatile unbinned goodness-of-fit procedure expressed by some people at the conference seems somewhat naive. A more important practical question is how to design a powerful goodness-of-fit test for each individual problem. It is not possible to answer this question unless we specify in more narrow terms the problem that we are trying to solve.